Sensitivity and specificity of information criteria

Choosing a model with too few parameters can involve making unrealistically simple assumptions and lead to high bias, poor prediction, and missed opportunities for insight. Such models are not flexible enough to describe the sample or the population well. A model with too many parameters can fit the observed data very well, but be too closely tailored to it. Such models may generalize poorly. Penalized-likelihood information criteria, such as Akaike’s Information Criterion (AIC), the Bayesian Information Criterion (BIC), the Consistent AIC, and the Adjusted BIC, are widely used for model selection. However, different criteria sometimes support different models, leading to uncertainty about which criterion is the most trustworthy. In some simple cases the comparison of two models using information criteria can be viewed as equivalent to a likelihood ratio test, with the different models representing different alpha levels (i.e., different emphases on sensitivity or specificity; Lin & Dayton 1997). This perspective may lead to insights about how to interpret the criteria in less simple situations. For example, AIC or BIC could be preferable, depending on sample size and on the relative importance one assigns to sensitivity versus specificity. Understanding the differences among the criteria may make it easier to compare their results and to use them to make informed decisions. INFORMATION CRITERIA 2 Many model selection techniques have been proposed in different settings (see, e.g., Miller, 2002; Pitt & Myung, 2002; Zucchini, 2000). Among other considerations, researchers must balance sensitivity (having enough parameters to adequately model the relationships among variables in the population) with specificity (not overfitting a model or suggesting nonexistent relationships). Several of the simplest and most common model selection criteria can be discussed in a unified way as log-likelihood functions with simple penalties. These include Akaike’s Information Criterion (AIC; Akaike, 1973), the Bayesian Information Criterion (BIC; Schwarz, 1978), Bozdogan’s consistent AIC (CAIC; Bozdogan, 1987), and the adjusted BIC (see Sclove, 1987). They consist of a goodness-of-fit term plus a penalty to control overfitting, and provide a standardized way to balance sensitivity and specificity. Each of these simple criteria involves choosing the model with the best penalized loglikelihood (i.e., the highest value of `−Anp where ` is the log-likelihood, An is some constant or some function of the sample size n, and p is the number of parameters in the model). For historical reasons, instead of finding the highest value of ` minus a penalty, this is sometimes expressed as finding the lowest value of −2` plus a penalty, i.e.,